------------------------------------------------------------
------------------------------------------------------------
Replication Materials for: Political Censorship in Large Language Models Originating from China

Authors:
- Jennifer Pan, Stanford University
- Xu Xu, Princeton University

------------------------------------------------------------
Citation
------------------------------------------------------------

If you use these materials, please cite: Pan, Jennifer and Xu Xu. 2026. "Political Censorship in Large Language Models Originating from China." PNAS Nexus.


------------------------------------------------------------
Computing Environment
------------------------------------------------------------
Operating System       : macOS Tahoe 26.2
Processor              : Apple M1 Max
Memory (RAM)           : 64 GB
System Type            : 64-bit operating system, ARM64-based processor



------------------------------------------------------------
Software
------------------------------------------------------------
R Version              : 4.5.2 (2025-10-31)
RStudio Version        : 2025.09.2+418
Python 3 (for data pre-processing):
- pandas
- fuzzywuzzy (for var_coding.py only)


------------------------------------------------------------
Overview
------------------------------------------------------------

This replication package contains:

1. Raw data (/data/raw): Complete LLM responses for all prompts across iterations in Chinese and English translated to Chinese
2. Intermediate data (/data/intermediate): data pre-processed from raw responses for main results
3. Accuracy data (/data/accuracy): accuracy data pre-processed from raw responses for Chinese responses
4. Pre-processing code (/code): Python scripts to generate intermediate and accuracy data
5. Analysis code (/code): R script to generate figures and tables from the paper based on intermediate data
6. Prompts (/data): Excel file documenting all prompts used in the study

We recommend users start by running the R analysis code based on the intermediate and accuracy data (fastest). Users can also regenerate intermediate data and accuracy data from raw data using Python scripts, but need to set directories and install packages appropriately. Users can also examine raw responses directly for alternative analyses.


------------------------------------------------------------
Instructions
------------------------------------------------------------


REPLICATE RESULTS (Using Pre-Processed Data):
------------------------------------------------------------
1. Install required R packages (see below)
2. Run R scripts in "code_r" folder to generate figures and tables


RE-GENERATE INTERMEDIATE/ACCURACY DATA (From Raw Data):
------------------------------------------------------------
1. Install Python packages: e.g., pandas, fuzzywuzzy
2. Run processing scripts in order:
   a. var_coding.py (generates intermediate data from raw data)
   b. accuracy.py (generates accuracy data from raw data for Chinese prompts)


R Package Installation:
------------------------------------------------------------
Required packages: dplyr (v1.1.4), ggplot2 (v3.5.2), tidyr (v1.3.1), 
                  stringr (v1.5.1), stargazer (v5.2.3), ggrepel (v0.9.6)

Use the following R code snippet:
------------------------------------------------------------
install_all <- function(packages) {
  installed_pkgs <- installed.packages()[, "Package"]
  for (pkg in packages) {
    if (!pkg %in% installed_pkgs) {
      install.packages(pkg, repos = 'http://cran.us.r-project.org')
    }
  }
}
packages <- c("dplyr", "ggplot2", "tidyr", "stringr", "stargazer",
              "ggrepel")
install_all(packages)
------------------------------------------------------------


------------------------------------------------------------
Folder Structure
------------------------------------------------------------

code/               - R script for replicating all figures and tables (replication.R)
                        - Python scripts for processing raw data (var_coding.py, accuracy.py)
data/                - Data files organized in 3 subdirectories, as well as 
                          excel file with prompts (prompts.xlsx),   
                          ref file for accuracy coding (chn_prompt_ref_entity_cn.csv, and 
                          ref file of categories for Figure S3 (infotype2.csv)
  raw/                - Raw LLM API responses (*_combined.csv)
  intermediate/  - Data for main results from raw responses (*.csv)
  accuracy/       - Accuracy data for main results from raw responses (*_combined_accuracy.csv)
graph/              - Figures generated by R script
table/               - Tables generated by R script


------------------------------------------------------------
Data Description
------------------------------------------------------------

File Naming Convention:
------------------------------------------------------------
censorship_[MODEL]_temp[TEMPERATURE]_[LANGUAGE]_[SUFFIX].csv

Where:
- [MODEL]               : LLM model tested (e.g., deepseek, baichuan13b, llama2)
- [TEMPERATURE] : model temperature setting (e.g., 0.3, 0.01)
- [LANGUAGE]        : Language of prompts and output
                                   chn = Chinese prompt and output
                                   engtrans = English prompt and output translated to Chinese for comparison
                                   chnengtrans = Chinese prompts that resulted in English output, translated back to 
                                                           Chinese (only relevant for llama models)
- [SUFFIX]               : Data type indicator
                                 _combined = raw data
                                (no suffix) = intermediate data for main results replication
                                _combined_accuracy = accuracy data

Examples:
- censorship_ernie_api_temp0.01_chn_combined.csv (raw)
- censorship_ernie_api_temp0.01_chn.csv (intermediate)
- censorship_ernie_api_temp0.01_chn_combined_accuracy.csv (accuracy)


------------------------------------------------------------
Variable Descriptions
------------------------------------------------------------


Raw data files (/data/raw)
------------------------------------------------------------

Variables:
- qn_num        : Question number (unique identifier for each prompt, 1 to 145)
- convX           : Response from conversation iteration X (1 up to 100)
- qnx_idx         : Question number (unique identifier for each prompt)
- prompt_txt    : Full text of the prompt sent to the LLM

Notes:
- Each row represents one prompt tested across all conversation iterations
- Some responses may contain special tags (e.g., <think>) that are
  stripped during intermediate data and accuracy data processing to represent what users see


Intermediate data files (/data/intermediate)
------------------------------------------------------------

Variables:
- qn_num                           : Question number (unique identifier for each prompt, 1 to 145)
- prompt_txt                       : Full text of the prompt sent to the LLM
- convX_len                       : Character length of response in conversation X (1 up to 100)
- modal_response             : The most frequently occurring response text across
                                            all conversations (or "unique" if all different)
- modal_response_conv    : Which conversation(s) produced the modal response (e.g., "conv1")
- longest_response            : The longest response text by character count across all conversations
- longest_response_conv  : Which conversation produced the longest response (e.g., "conv9")
- per_nonresponse            : Percentage of conversations that resulted in a refusal to respond
- conv_nonresponse          : List of specific conversations that resulted in non-responses
- per_exact_prompt            : Percentage of conversations where response exact match of prompt text
- per_fuzzy_prompt            : Percentage of conversations where response had high similarity to prompt


Accuracy data files (/data/accuracy)
------------------------------------------------------------
Note: Accuracy files are only generated for Chinese language prompts (_chn).

Variables:
- qns_idx             : Question index (corresponds to qn_num-1, 0 to 144)
- convX               : Accuracy assessment for conversation X (1 up to 100)
- nonresp_per     : Proportion of conversations coded as "refusal to respond"
- inaccurate_per  : Proportion of responses (excluding refusals) coded as "completely inaccurate"
